3d radiomic platform for imaging biomarker development

ABSTRACT

A platform is provided for generating 3D models of a tumor segmented from a series of 2D medical images and for identifying from these 3D models, radiomic features that may be used for diagnostic, prognostic, and treatment response assessment of the tumor. The radiomic features may be shape-based features, intensity-based features, textural features, and filter-based features. The radiomic features are compared to remove sufficiently redundant features, thereby producing a reduced set of radiomic features, which is then compared to separate genomic data and/or outcome data to identify clinically and biologically significant radiomic features for diagnostic, prognostic, and treatment response assessment, other applications.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims priority to U.S. Provisional Application No.62/693,371, filed Jul. 2, 2018, which is incorporated herein byreference in its entirety.

FIELD OF THE INVENTION

The present disclosure relates to examining medical images of a tumorsand surrounding tissue and, more particularly, to developingthree-dimensional models of radiomic features of the tumor for 3D imagebiomarker development.

BACKGROUND

The background description provided herein is for the purpose ofgenerally presenting the context of the disclosure. Work of thepresently named inventors, to the extent it is described in thisbackground section, as well as aspects of the description that may nototherwise qualify as prior art at the time of filing, are neitherexpressly nor impliedly admitted as prior art against the presentdisclosure.

Accurate assessment of treatment response provides clinicians withvaluable information about the efficacy of treatment. However, currentassessment techniques are often subjective and inconsistent. For manytechniques, a clinician collects two-dimensional images of a tumor atdifferent stages of treatment. The clinician then analyses the imagesand determines tumor growth/reduction using a one dimensional measure,i.e., tumor longest diameter. In some examples, clinicians usecomputer-based systems to identify tumors and track changes in tumorsize using these two-dimensional images. Yet, because these conventionalsystems examine only two-dimensional image data, the systems, whileoffering a level of automation, often fail to accurately track truevolumetric changes in the tumor, and thereby, they often fail to provideaccurate information about treatment efficacy.

SUMMARY OF THE INVENTION

The present application presents a platform for identifyingthree-dimensional (3D) radiomic features for use in examining medicalimage data for subjects. In particular the platform may be used foridentifying 3D radiomic features for developing imaging biomarker forvarious pathologies, including cancers. The present techniques provide aquantitative and consistent way to structure medical image data, inparticular radiology data, and standardize response data collection.

The present techniques provide automated processes capable of examininglarge volumes of stored, digital radiology data, in an optimized mannerthat reduces image processing burdens and that expands diagnostic andprognostic and disease monitoring accuracy by identifying from among alarge cohort of radiomic features, those particular radiomic featuresthat are most correlative from a diagnostic and prognostic and diseasemonitoring viewpoint. The result is not only that large databases ofradiology data may be examined to generate a reduced set of particularlyimportant image biomarkers, but the responsiveness of tumors totreatment is more accurately assessed and in a more processing efficientmanner.

As shown, the platform provides objective and standardize responseassessment. The platform is able to assess target tissue and tumorimaging patterns not recognizable to clinicians examining images usingthe naked eye or using 2D images. Indeed, the platform is able togenerate and assess entirely new imaging patterns, e.g., new 3D radiomicpatterns. As such, the platform is able to identify imaging featuresthat can improve prognosis (or clinical outcome prediction) and imagingfeatures associated with genomic data. In these ways, the platform isable develop non-invasive imaging biomarkers for genomic data.

In accordance with an example, a computer-implemented method to analyzemedical image data, the method comprises: obtaining, at one or moreprocessors, the medical image data comprising a plurality oftwo-dimensional (2D) medical images; performing target tissue detectionand target tissue segmentation for each 2D medical image to produce aset of segmented target tissue images; generating a three-dimensional(3D) model of detected and segmented target tissue; identifying a masterset of radiomic features for the 3D model of detected and segmentedtarget tissue; comparing at least some of the radiomic features in themaster set to identify redundant radiomic features for the 3D model ofdetected and segmented target tissue; excluding the redundant radiomicfeatures from the 3D model of detected and segmented target tissue; andextracting a selected set of radiomic features from the 3D model ofdetected and segmented target tissue.

In some examples, the target tissue is tumor tissue. In some examples,the target tissue is tumor tissue and surrounding tissue.

In some examples, the method includes identifying from among theselected set of radiomic features, radiomic features that aresignificantly associated with clinical outcomes and/or genomic data.

In accordance with another example, a computing device, having one ormore processors is, configured to obtain, at one or more processors, themedical image data comprising a plurality of two-dimensional (2D)medical images; perform target tissue detection and target tissuesegmentation for each 2D medical image to produce a set of segmentedtarget tissue images; generate three-dimensional (3D) model of detectedand segmented target tissue; identify a master set of radiomic featuresfor the 3D model of detected and segmented target tissue; compare atleast some of the radiomic features in the master set to identifyredundant radiomic features for the 3D model of detected and segmentedtarget tissue; exclude the redundant radiomic features from the 3D modelof detected and segmented target tissue; and extract a selected set ofradiomic features from the 3D model of detected and segmented targettissue.

In some examples, the selected set of radiomic features is analyzed todetermine tumor diagnosis or prognosis.

In some examples, the selected set of radiomic features are analyzed todetermine tumor treatment effectiveness, and normal tissue side effectincluding for example changes to the selected set of radiomic featuresin response to tumor treatments.

In some examples, a 3D graphic that visually depicts one or more of theselected set of radiomic features is generated and provided to medicalprofessionals, for example, using a graphical user interface display.

In some examples, downloadable radiomic feature files, in csv(Comma-Separated Values) format, are provided to medical professionalsby the system.

In some examples, downloadable segmentation files in DICOM (DigitalImaging and Communications in Medicine), nrrd (Nearly Raw Raster Data),or nifti (Neuroimaging Informatics Technology Initiative) format arealso provided to medical professionals by the system.

In some examples, the selected set of radiomic features are determinedby comparing a master set of radiomic features to genomic data and/or topatient treatment outcome data to identify which radiomic featureswithin the master set of features are indicative of a tumor diagnosis,tumor prognosis, tumor treatment effectiveness, the effectiveness oftreatment on normal tissues, etc.

In some examples, the radiomic features are imaging data that correlateto and are indicators of molecular or genomic data, treatment responsedata, diagnostic data, prognostic data, and/or classifiers of cancerpatient risk stratification.

In some examples, the radiomic features are shape-based features (e.g.,tumor longest diameter, 3D volume, surface area, sphericity, surfacesmoothness, numbers of voxels, etc.).

In some examples, the radiomic features are Intensity-based features(e.g., average tumor intensity, skewness of tumor intensitydistribution, and kurtosis of tumor intensity distribution).

In some examples, the radiomic features are textural features (e.g.,contrast, correlation, and homogeneity).

In some examples, the radiomic features are filter-based features (e.g.,wavelets and Laplacian of Gaussian filters).

In some examples, normalization is performed on the medical imagesbefore the 3D model is generated, for example, to ensure a normalizationof image intensity distribution, image color, and voxel size for the 3Dmodel.

In some examples, the processes herein may be performed before a tumortherapy and again after a tumor therapy, or multiple times after a tumortherapy. For example a 3D model of detected and segmented target tissuemay be generated from medial images taken before tumor therapy andgenerated again from medical images taken after the tumor therapy. Adetermined selected set of radiomic features from the 3D model may thenbe extracted and compared before tumor therapy and after tumor therapy,and the changes in each (or at least one) of the radiomic featuresquantified. The radiomic features compared before and after therapy maybe any of those discussed herein and others, including shape-basedfeatures, intensity-based features, textural features, and filter-basedfeatures

In some examples, the comparison of 3D models before and after tumortherapy, e.g., the comparison of radiomic features before and aftertumor therapy, is used by a processing system to determine the efficacyof the therapy (e.g., whether there is a reduction in tumor longestdiameter, 3D volume, or surface area, or a change in sphericity, surfacesmoothness, numbers of voxels, etc., or a change in average tumorintensity, skewness of tumor intensity distribution, or kurtosis oftumor intensity distribution, or a change in contrast, correlation, andhomogeneity, or a change in wavelets and Laplacian of Gaussian filters.In some examples, that comparison data is used by a processing system todetermine a next therapy or a group of matched therapies, based on thedetermined efficacy (e.g., based on radiomic feature significantlyassociated with clinical outcomes and/or genomic data and the amount ofchanges in those radiomic features).

BRIEF DESCRIPTION OF THE DRAWINGS

This patent or application file contains at least one drawing executedin color. Copies of this patent or patent application publication withcolor drawing(s) will be provided by the United States Patent andTrademark Office upon request and payment of the necessary fee.

The figures described below depict various aspects of the system andmethods disclosed herein. It should be understood that each figuredepicts an example of aspects of the present systems and methods.

FIG. 1 is a schematic illustration of an example computer processingsystem for analyzing radiology images, such as tumor images (and othertarget tissue images), to generate a 3D model and to generate 3Dradiomic features of regions of interest in the images

FIG. 2 illustrates an example computing device for implementing thesystems of FIG. 1 and the processes of FIGS. 3, 8, and 9, in accordancewith an example.

FIG. 3 is a block diagram of an example method for performing 3Dradiomic feature generation and analysis on radiology images, as may beperformed by the processing system of FIG. 1 and/or the computing deviceof FIG. 2, in accordance with an example.

FIG. 4 illustrates 2D medical images of a portion of a tumor taken atdifferent times in the process of treating a subject.

FIG. 5 illustrates 3D models of a tumor for a patient generated from 2Dmedical images taken at the different times of FIG. 4.

FIG. 6 illustrates 2D medical images taken of a portion of a tumorshowing inconclusive differences in 2D features, such as major and minoraxes.

FIG. 7 is a plot of the changes in a 3D radiomic feature (e.g., tumorvolume) developed from the 3D models of FIG. 5, illustrating a clearcorrelation between changes in the 3D radiomic feature and treatmentstages.

FIG. 8 is a block diagram of an example process for redundant radiomicfeature identification and exclusion, as may be implemented by themethod of FIG. 3, in accordance with an example.

FIG. 9 is a block diagram of an example process for identification of adiagnostic radiomic feature set, as may be implemented by the method ofFIG. 3, in accordance with an example.

FIG. 10 is a plot showing of c-index (concordance) of clinical data,molecular data, and radiomic feature data for an example implementation.As shown, integrating radiomic feature data developed herein, withclinical and molecular data shows an increase in predictive power (i.e.,a higher c-index) of outcome data.

DETAILED DESCRIPTION

The present application presents a platform for identifying radiomicfeatures for use in examining medical image data for subjects. Inparticular, the platform may be used for identifying radiomic featuresfor developing imaging biomarker for various pathologies, includingcancers. These radiomic features are generated from a three-dimensional(3D) model of a target tissue, where that 3D model has been generatedfrom a series of 2D medical images.

The present techniques provide a quantitative and consistent way tostructure medical image data (e.g., radiology data) and to standardizeresponse data collection. The present techniques provide automatedprocesses capable of examining large volumes of stored, digitalradiology data, in an optimized manner that reduces image processingburdens and that expands diagnostic accuracy by identifying from among alarge cohort of radiomic features, those particular radiomic featuresthat are most correlative from a diagnostic viewpoint. The result is notonly that large databases of radiology data may be examined to generatea reduced set of particularly important image biomarkers, but theresponsiveness of tumors to treatment is more accurately assessed, andin shorter period of time.

The present techniques streamline the therapeutic assessment process,making the process quantitative, more objective and consistent.Furthermore, generated radiomic features can capture meaningfulclinical, biological, and anisotropic changes of tumor over the courseof treatment. These changes are not identifiable from traditionaltwo-dimensional imaging and analysis and thereby provide new indicators(which we also term image biomarkers) of tumor diagnosis and prognosis.Further still, the techniques herein include segmentation andnormalization techniques to generate a structure data set from numerousdisparate medical image databases.

Additionally, the present techniques apply pair-wise comparisons betweenradiomic features to identify redundant features that may be excludedwhen modelling and analyzing the tumor. Redundant feature extraction cansubstantially reduce the computer processing resources needed to performtumor analysis. We have found that redundant feature exclusion has beenable to reduce a master radiomic feature set from numbering in the 1000sto identifying a selected group of less than 100 radiomic features andin some instances less than 10 radiomic features that may be used fordiagnostic and prognostic assessment.

The tumors that may be examined by the present techniques includemalignant and benign tumors. Example malignant tumors may includevarious cancer types, such as, breast cancer, colon cancer, gastriccancer, endometrium cancer, ovarian cancer, hepatobiliary tract cancer,urinary tract cancer, lung cancer, brain cancer, and skin cancers. Thepresent techniques may be used to develop 3D radiomic features thatcorrelate to any of these or other tumor types. Moreover, the presenttechniques may be used to develop a 3D model of radiomic features thatact as 3D image biomarkers of the various tumor types. In some examples,the 3D model of radiomic features is constructed by analyzingintra-tumor radiomic features, that is, radiomic features of differenttumor types within a subject or across subjects. In some examples, the3D model of radiomic features is constructed by analyzing inter-tumorradiomic features, that is, radiomic features of the same tumor type,but appearing in different locations within a subject.

FIG. 1 illustrates a computer processing system 100 for analyzingradiology images, such as tumor images (and other target tissue images)to generate a 3D model and to generate 3D radiomic features of regionsof interest in the images.

The processing system 100 includes an radiomic feature processingframework 102 communicatively coupled to a network 106 for receivingmedical images and other data genomic data and drug treatment andpatient outcome data. For example, the network 106 is shown coupled to avariety of different sources, from a variety of different sources,including (i) medical imaging databases of healthcare providers(Provider_1 104) such as hospitals, physician groups, labs, etc.; (ii)dedicated digital medical image scanners 108 that may be, by way ofexample, any suitable optical histopathology slide scanner including 20×and 40× resolution magnification scanners; (iii) medical imagerepositories 110 such as databases of stored medical images and (iii)the Cancer Imaging Archive (TCIA). Each of the image sources may presentmultiple image sources. In the illustrated example, the Provider_1 104and the medical images repository 110 may include genomic data,treatment data, and/or patient outcome data, as desired. The processingsystem 100 may be coupled to other medical data sources (not shown), aswell.

The medical image data herein may be any suitable two-dimensionalradiology images. The radiology images may be from any suitable imagingmodality, examples of which include X-ray images, computed tomography(CT) scans, magnetic resonance imaging (MRI), nuclear medicine imaging(NMR), positron-emission tomography (PET), etc.

The processing system 100 may be implemented on a computing device suchas a computer, tablet or other mobile computing device, or server. Thesystem 100 may include a number of processors, controllers or otherelectronic components for processing or facilitating the image capture,generation, or storage and image analysis, as described herein. Anexample computing device 200 for implementing processing system 100 isillustrated in FIG. 2.

As illustrated in FIG. 2, the system 100 may be implemented on thecomputing device 200 and in particular on one or more processing units,which may represent Central Processing Units (CPUs), and/or on one ormore or Graphical Processing Units (GPUs), including clusters of CPUsand/or GPUs. Features and functions described for the system 100 may bestored on and implemented from one or more non-transitorycomputer-readable media of the computing device 200. Thecomputer-readable media may include, for example, an operating systemand a radiomic feature platform corresponding to elements of theradiomic feature processing framework 102. The radiomic feature platformmay include a segmentation controller, a 3D model generator, a medicalimage normalization controller, a radiomic feature extractioncontroller, a redundancy exclusion controller, and a selected radiomicfeatures generator.

More generally, the computer-readable media may store trained 3D models,a master set of radiomic features, an identification of redundantradiomic features, and selected radiomic features.

The selected radiomic features generated by the system 100 are a subsetof radiomic features optimally-defined for diagnostic and prognosticassessment of target tissue, including, for example, tumor andsurrounding tissue.

These selected radiomic features may be generated from a 3D model oftarget tissue.

These selected radiomic features, for example, may include imaging datathat correlates to and thereby indicates particular molecular data,genomic data, treatment response data, diagnostic data, prognostic data,and/or classifiers of cancer patient risk stratification. In this way,the selected radiomic features can provide imaging biomarkers.

The selected radiomic features generated by the system 100 may include:shape-based features (e.g., tumor longest diameter, 3D volume, surfacearea, sphericity, surface smoothness, numbers of voxels, etc.);intensity-based features (e.g., average tumor intensity, skewness oftumor intensity distribution, and kurtosis of tumor intensitydistribution); textural features (e.g., contrast, correlation, andhomogeneity); and filter-based features (e.g., wavelets and Laplacian ofGaussian filters).

The computing device 200 includes a network interface communicativelycoupled to the network 106, for communicating to and/or from a portablepersonal computer, smart phone, electronic document, tablet, and/ordesktop personal computer, or other computing devices. The computingdevice further includes an I/O interface connected to devices, such asdigital displays, user input devices, etc. In the illustrated example,the processing system 100 is implemented on a single server 200.However, the functions of the system 100 may be implemented acrossdistributed devices 200, 202, 204, etc. connected to one another througha communication link. In other examples, functionality of the system 100may be distributed across any number of devices, including the portablepersonal computer, smart phone, electronic document, tablet, and desktoppersonal computer devices shown. The network 106 may be a public networksuch as the Internet, private network such as research institutions orcorporations private network, or any combination thereof. Networks caninclude, local area network (LAN), wide area network (WAN), cellular,satellite, or other network infrastructure, whether wireless or wired.The network can utilize communications protocols, including packet-basedand/or datagram-based protocols such as internet protocol (IP),transmission control protocol (TCP), user datagram protocol (UDP), orother types of protocols. Moreover, the network can include a number ofdevices that facilitate network communications and/or form a hardwarebasis for the networks, such as switches, routers, gateways, accesspoints (such as a wireless access point as shown), firewalls, basestations, repeaters, backbone devices, etc.

The computer-readable media may include executable computer-readablecode stored thereon for programming a computer (e.g., comprising aprocessor(s) and GPU(s)) to the techniques herein. Examples of suchcomputer-readable storage media include a hard disk, a CD-ROM, digitalversatile disks (DVDs), an optical storage device, a magnetic storagedevice, a ROM (Read Only Memory), a PROM (Programmable Read OnlyMemory), an EPROM (Erasable Programmable Read Only Memory), an EEPROM(Electrically Erasable Programmable Read Only Memory) and a Flashmemory. More generally, the processing units of the computing device 200may represent a CPU-type processing unit, a GPU-type processing unit, afield-programmable gate array (FPGA), another class of digital signalprocessor (DSP), or other hardware logic components that can be drivenby a CPU.

Returning to FIG. 1, the radiomic feature processing framework 102includes a segmentation module that automatically identifies targettissue, such as tumors, in received medical images, segments that targettissue for analysis by the framework 102. In some examples, thesegmentation module may implement a convolutional neural network toperform whole slide image segmentation. More generally, any number ofunsupervised or supervised methods of image segmentation may be used.The segmentation module may be configured to perform segmentation oneach 2D medical image, thereby removing from the image and retainingthose portions corresponding to the target tissue, e.g., tumor. Thesegmentation may be patch segmentation, in which groups of pixels areexamined to identify the target tissue within the medical image. Forexample, a segmentation process may identify a particular geometricpatch that will be used for analyzing image data. Patches may begeometric, e.g., a repeating pattern of square or rectangular pixelsdefined across each medical image and at a pixel size sufficient toanalyze changes in topology and morphology in medical images. Examplepatch sizes include 256×256 pixels, although fewer pixels can be used,such as 90×90, 80×80, 70×70, 60×60, 50×50, and so on, down to at least10×10, and even further, such as 5×5 depending on the application. Inother examples, patch type may be non-geometric, also termed herein a“super pixel,” where each patch is allowed to vary in shape, but wherethe super pixels are generated to include a sufficient threshold ofimaging information for topology and morphology analysis.

A normalization module normalizes the segmented image data. Thenormalization may normalize pixel or voxel intensity, pixel or voxelcolor, or other factors. The normalization module may normalize medicalimages from different medical image sources, thereby allowing uniformanalysis across a large combined database of medical images. In someconfigurations, the normalization module is configured to normalizemedical image data across different imaging modalities, for example,normalizing segmented X-Ray image data with segmented MRI data.

The normalization module may perform normalization on 2D medical images,in some examples. While in other examples, the normalization module mayperform normalization on a 3D model generated by the framework 102, asdescribed herein.

A 3D model generator module of the framework 102 stores the segmentedand normalized image data and performs registration and data stackingprocesses to construct a 3D model of the target tissue or tumor. Aradiomic feature processing module analyses the resulting 3D model andidentifies radiomic features from that 3D model, redundant radiomicfeatures for exclusion, and a resulting subset of clinically andbiologically meaningful radiomic features (also termed diagnosticradiomic features) that can be used to diagnose tumors and/or indicatetumor state.

FIG. 3 illustrates a process 300 for performing radiomic featuregeneration and analysis on radiology images, as may be performed by theprocessing system 100, e.g., as implemented by the computing device 200.Radiology images from image sources are imported and tumor locationdetection is performed. In an example automated configuration, tumorlocation detection is performed by comparing received radiology imagesto a database of tumor-identified medical images, e.g., through the useof a convolution neural network. The convolution neural network may betrained by clinical and/or synthetic training medical images, includingmedical images of tissue samples corresponding to the tumor to bedetected, of tissue samples that do not exhibit the tumor to bedetected, of tissue samples that exhibit other tumor types, and/ortissue samples exhibiting no tumor. It is noted, that in a 2D medicalimage, the process 300 may identify a number of different locations oftumors, as well as any number of different tumor types that may bepresent in the 2D medical image.

While the process 300 is discussed in reference to the particularexample of tumor location detection, more broadly, the process 300 maybe used to identify any target tissue. In some examples, that targettissue may encompass a tumor and surrounding tissue. In other examples,the target tissue may be any tissue imaged in the 2D images, such asorgan tissue or bone tissue. That is, the segmentation, normalization,3D modelling, and other processes described herein as applied to tumorregions may be applied to any target tissue under examination.

With the tumor location detection complete, tumor border segmentationmay be achieved, for example, using an edge detection based segmentationtechnique. The tumor or tumor regions in the medical image are segmentedout and a segmentation confirmation process may be optionallyimplemented. Segmentation confirmation may include, for example,assessing the shape of the tumor border to identify any regions of largeshifts in intensity pixel to pixel that may indicate an impropersegmentation. Segmentation confirmation may be based on the number ofsuch regions of large shift across the entire image or across a portionof the image, for example. If the segmentation confirmation is returnedas below a threshold confirmation level, then the process may repeat,until a sufficiently accurate segmentation has been achieved.

Each segmented medical image is a 2D image of a tumor, or target tissue.The segmented medical images are buffered and stored by a 3D tumor modelgenerator that constructs a 3D model from the segmented 2D images.

In some examples, image intensity distribution and voxel sizenormalization is performed on the 3D model to provide uniforminformation across the 3D model for radiomic feature extraction. Asdiscussed herein, such normalization allows for compensating indifferences between different medical image sources (such as differentmedical imaging scanners), as well as across entirely different medicalimaging modalities.

FIG. 4 illustrates example 2D medical image data provided to the process300 and comprising 2D MRI images of a brain scan taken at five (5),different points in time. Time1 and Time2 are pre-treatment scans. Braintumors are identified in the medical images. Time3-Time5 are medicalimages taken after surgery to remove a portion of the brain tumor andshow the regression of tumor in response to therapy. FIG. 4 illustrates2D T1 structure images and 2D T2 FLAIR (fluid attenuated inversionrecovery) images, either or both of which may be provided to the process300.

FIG. 5 illustrates example 3D models generated for each of the differentimage capture times of FIG. 4, where the 3D models have been generatedby the process 300 from multiple 2D images taken at each time,Time1-Time5.

Returning to FIG. 3, with the 3D model normalized, a master set ofradiomic features are extracted from the 3D model. The radiomic featuresmay include, by way of example, shape-based features (e.g., tumorlongest diameter, 3D volume, surface area, sphericity, surfacesmoothness, numbers of voxels, etc.), intensity-based features (e.g.,average tumor intensity, skewness of tumor intensity distribution, andkurtosis of tumor intensity distribution), textural features (e.g.,contrast, correlation, and homogeneity); and filter-based features(e.g., wavelets and Laplacian of Gaussian filters).

An example radiomic feature that may be extracted from the 3D model isthe tumor 3D volume. FIG. 6 shows a 2D medical image of the tumorshowing major and minor (i.e., long and short) axes of the tumor region,at Time2, before surgery, and at Time3, after surgery. As indicated,just looking at this 2D image and examining for changes in the major orminor axes is inconclusive. This 2D image alone does not sufficientlyinstruct a clinician on the successfulness of the surgery.

In contrast, when we examined the tumor 3D volume radiomic featuregenerated from the process 300, a clear difference in volume was shown.FIG. 7 plots the measures of the tumor 3D volume radiomic feature, as afunction of time, in an example. The change in 3D volume clearlycorrelates with surgery. Indeed, changes in the 3D volume radiomicfeature of the tumor further correlated with subsequent, post-surgerytreatment. These indications from the 3D volume stand in stark contrastto the inconclusive results using 2D medical image data only (i.e., FIG.6). In other examples, changes in radiomic features may be comparedbefore and after other kinds of treatment events. For example, the tumor3D volume radiomic feature generated from the process 300 may begenerated before a patient receives a therapy and again after a patientreceives a therapy. As another example, the tumor 3D volume radiomicfeature generated from the process 300 may be generated more than onceafter a patient receives a therapy. Exemplary therapies includechemotherapy, radiation, immunotherapy, PARP inhibitors, CAR T-celltherapy, and cancer vaccines, among other therapies. In an example, thetumor 3D volume radiomic feature generated from the process 300 may begenerated in the first 60 days after receipt of a therapy. In someexamples, the tumor 3D volume radiomic feature generated from theprocess 300 that is generated after receipt of a therapy may be utilizedto determine the efficacy of the therapy. In some examples, the tumor 3Dvolume radiomic feature generated from the process 300 may be utilizedto determine a next therapy.

Returning to FIG. 3, the process 300 further includes a redundantradiomic feature exclusion process, discussed in further detail in theexample implementation of FIG. 8, and a diagnostic radiomic featureextraction process, discussed in further detail in the exampleimplementation of FIG. 9.

FIG. 8 illustrates a process 400 for identifying redundant radiomicfeatures from among a master set of radiomic features identified in theprocess 300. The master set of extracted radiomic features (e.g., 500,1000, 2000, or more) is provided. For each of the features in the masterset, pair-wise correlations are determined for each other of thefeatures in the master set. The pair-wise correlations are scored andany correlations with a score above a threshold value (e.g.,correlations>0.95) are identified, correlation scores are tallied forthe radiomic features. Those radiomic features with a high averagecorrelation score are removed from the masters set to generate aselected radiomic features set. The process then repeats another rounduntil the radiomic features that are left are features that havepair-wise correlations below the threshold (e.g., <0.95). After thefinal round, a subset of radiomic features is produced by the process400.

In FIG. 9, a process 500 obtains the subset of radiomic features fromprocess 400 and determines a further subset of radiomic features, inparticular, a selected radiomic feature set that may be used fordiagnostic or prognostic assessments, for example to assess tumor type,treatment efficacy, etc.

In an example implementation of the process 500, the radiomic features,genomics data corresponding to the obtained medical images, and patienttreatment outcome data are provided to a univariate analysis processor.The analysis processor compares each of the radiomic features obtainedfrom the process 400 to categorized genomics data and/or to categorizedoutcome data. This comparison is done across a large database of medicalimages and 3D models. The process 500 determines, from the largedatabase comparisons, whether any (and which) of the radiomic featuresare significantly associated with either genomics data or outcome data,and those significant associations are stored in a database format toidentify the radiomic feature and the associations of that feature. Theradiomic features identified as having significant associations are thendetermined to be clinically and/or biologically significant and arestored as a selected radiomic feature set. The process 500 may thengenerate an enhanced 3D model exhibiting only the radiomic featuresidentified as having significant associations. In some examples,radiomic features having significant associations may be illustrated inthe previously generated 3D model (3D tumor model generation process ofFIG. 3) using a segmentation and color coding to visually indicate thoseradiomic features. In some examples, the process 500 generates a digitalreport or data file that includes a listing of the radiomic featureshaving significant associations, a scoring of the level of significance(e.g., a p-value) and the associated clinical data and/or biologicaldata. In some examples, the significant associations are covariant, inwhich case the report will include covariant associations.

As used herein “significant associations” refers to statisticallysignificant associations, assessed using a statistical model such as asurvival function analysis. Example statistical models include alog-rank test model or a Kaplan-Meier model. Others statistical modelsinclude a likelihood ratio test or a Wald test. Yet, in some examplesherein, including FIG. 10 discussed below, a regression model thatallows for covariant analysis is used, such as the Cox proportionalhazards regression analysis. As used herein significant associations,resulting from the model used, refers to associations having a p-valueof <0.05, <0.045, <0.040, <0.035, <0.030, <0.025, <0.020, <0.015,<0.010, <0.005, or <0.001.

As shown in FIG. 10, the radiomic feature data developed by the process300 may be combined with clinical data (e.g., age in the illustratedexample) and molecular data (e.g., MGMT methylation) to produce anintegrated radiomic-based indicator. This integrated radiomic-basedindicator can provide significantly improved c-index (Cox index of a Coxproportional hazards) values for assessing tumor and other target tissue(in the illustrated example, the c-index values shown all exhibit ap-value <0.005). That is, while radiomic features alone can serve asimaging biomarkers, in some cases, radiomic features may be combinedwith other data to provide even more accurate integrated imagingbiomarkers. FIG. 10 is a plot showing c-index (concordance) for clinicaldata, molecular data, and radiomic feature data, in an example.Integrating these data sets together, e.g., through multivariate Coxproportional hazards model, results in a substantially higher c-index,which can serve as a more accurate imaging biomarker. Molecular data mayinclude gene expression irregularities, irregular microRNA profile,splice variations, DNA methylation, etc.

Further still, the selected radiomic features may be cross-correlatedagainst clinical outcome data and genomic data to identify significantassociations. Clinical outcome data would include stored data ondisease-free survival, tumor stage, response rate to particulartherapeutics, absence of disease, quality of life, etc. Genomic data mayinclude variations in DNA structure or sequence, or copy number, etc.These data types may be stored in and received from databases, such aselectronic medical images databases/repositories, medical providerelectronic databases, and the like.

It is noted herein, the processing comparisons and analyses may beimplemented through trained or untrained processing techniques. Forexample, 2D image segmentation, 3D model normalization, redundantradiomic feature exclusion, pair-wise correlations, and determinationsof diagnostic radiomic features may be achieved using a deep learningframework or a plurality of deep learning frameworks, such as one foreach process. Broadly speaking, the deep learning framework mayimplement any suitable statistical model (e.g., a neural network orother model implemented through a machine learning process) that will beapplied to each of the received images and other data (e.g., genomic andoutcome data). As discussed herein, that statistical model may beimplemented in a variety of manners. In some examples, machine learningis used to evaluate training images and develop classifiers thatcorrelate predetermined image features to specific categories. Forexample, image features can be identified as training classifiers usinga learning algorithm such as Neural Network, Support Vector Machine(SVM) or other machine learning process. Once classifiers within thestatistical model are adequately trained with a series of trainingimages, the statistical model may be employed in real time to analyzesubsequent images provided as input to the statistical model foridentifying radiomic features and correlations. In some examples, whenstatistical model implemented using a neural network, the neural networkmay be configured in a variety of ways. In some examples, the neuralnetwork may be a deep neural network and/or a convolutional neuralnetwork. In some examples, the neural network can be a distributed andscalable neural network. The neural network may be customized in avariety of manners, including providing a specific top layer such as butnot limited to a logistics regression top layer. A convolutional neuralnetwork can be considered as a neural network that contains sets ofnodes with tied parameters. A deep convolutional neural network can beconsidered as having a stacked structure with a plurality of layers. Theneural network or other machine learning processes may include manydifferent sizes, numbers of layers and levels of connectedness. Somelayers can correspond to stacked convolutional layers (optionallyfollowed by contrast normalization and max-pooling) followed by one ormore fully-connected layers. For neural networks trained by largedatasets, the number of layers and layer size can be increased by usingdropout to address the potential problem of overfitting. In someinstances, a neural network can be designed to forego the use of fullyconnected upper layers at the top of the network. By forcing the networkto go through dimensionality reduction in middle layers, a neuralnetwork model can be designed that is quite deep, while dramaticallyreducing the number of learned parameters.

The 3D models, master set of radiomic features, selected set of radiomicfeatures, and correlations and significant associations determinedherein may then be visually indicated to a user, for example Provider_1,through a dashboard graphical user interface provided to a computermonitor or other display.

Throughout this specification, plural instances may implementcomponents, operations, or structures described as a single instance.Although individual operations of one or more methods are illustratedand described as separate operations, one or more of the individualoperations may be performed concurrently, and nothing requires that theoperations be performed in the order illustrated. Structures andfunctionality presented as separate components in example configurationsmay be implemented as a combined structure or component. Similarly,structures and functionality presented as a single component may beimplemented as separate components or multiple components. These andother variations, modifications, additions, and improvements fall withinthe scope of the subject matter herein.

Additionally, certain embodiments are described herein as includinglogic or a number of routines, subroutines, applications, orinstructions. These may constitute either software (e.g., code embodiedon a machine-readable medium or in a transmission signal) or hardware.In hardware, the routines, etc., are tangible units capable ofperforming certain operations and may be configured or arranged in acertain manner. In example embodiments, one or more computer systems(e.g., a standalone, client or server computer system) or one or morehardware modules of a computer system (e.g., a processor or a group ofprocessors) may be configured by software (e.g., an application orapplication portion) as a hardware module that operates to performcertain operations as described herein.

In various embodiments, a hardware module may be implementedmechanically or electronically. For example, a hardware module maycomprise dedicated circuitry or logic that is permanently configured(e.g., as a special-purpose processor, such as a microcontroller, fieldprogrammable gate array (FPGA) or an application-specific integratedcircuit (ASIC)) to perform certain operations. A hardware module mayalso comprise programmable logic or circuitry (e.g., as encompassedwithin a general-purpose processor or other programmable processor) thatis temporarily configured by software to perform certain operations. Itwill be appreciated that the decision to implement a hardware modulemechanically, in dedicated and permanently configured circuitry, or intemporarily configured circuitry (e.g., configured by software) may bedriven by cost and time considerations.

Accordingly, the term “hardware module” should be understood toencompass a tangible entity, be that an entity that is physicallyconstructed, permanently configured (e.g., hardwired), or temporarilyconfigured (e.g., programmed) to operate in a certain manner or toperform certain operations described herein. Considering embodiments inwhich hardware modules are temporarily configured (e.g., programmed),each of the hardware modules need not be configured or instantiated atany one instance in time. For example, where the hardware modulescomprise a general-purpose processor configured using software, thegeneral-purpose processor may be configured as respective differenthardware modules at different times. Software may accordingly configurea processor, for example, to constitute a particular hardware module atone instance of time and to constitute a different hardware module at adifferent instance of time.

Hardware modules can provide information to, and receive informationfrom, other hardware modules. Accordingly, the described hardwaremodules may be regarded as being communicatively coupled. Where multipleof such hardware modules exist contemporaneously, communications may beachieved through signal transmission (e.g., over appropriate circuitsand buses) that connects the hardware modules. In embodiments in whichmultiple hardware modules are configured or instantiated at differenttimes, communications between such hardware modules may be achieved, forexample, through the storage and retrieval of information in memorystructures to which the multiple hardware modules have access. Forexample, one hardware module may perform an operation and store theoutput of that operation in a memory device to which it iscommunicatively coupled. A further hardware module may then, at a latertime, access the memory device to retrieve and process the storedoutput. Hardware modules may also initiate communications with input oroutput devices, and can operate on a resource (e.g., a collection ofinformation).

The various operations of the example methods described herein can beperformed, at least partially, by one or more processors that aretemporarily configured (e.g., by software) or permanently configured toperform the relevant operations. Whether temporarily or permanentlyconfigured, such processors may constitute processor-implemented modulesthat operate to perform one or more operations or functions. The modulesreferred to herein may, in some example embodiments, compriseprocessor-implemented modules.

Similarly, the methods or routines described herein may be at leastpartially processor-implemented. For example, at least some of theoperations of a method can be performed by one or more processors orprocessor-implemented hardware modules. The performance of certain ofthe operations may be distributed among the one or more processors, notonly residing within a single machine, but also deployed across a numberof machines. In some example embodiments, the processor or processorsmay be located in a single location (e.g., within a home environment, anoffice environment or as a server farm), while in other embodiments theprocessors may be distributed across a number of locations.

The performance of certain of the operations may be distributed amongthe one or more processors, not only residing within a single machine,but also deployed across a number of machines. In some exampleembodiments, the one or more processors or processor-implemented modulesmay be located in a single geographic location (e.g., within a homeenvironment, an office environment, or a server farm). In other exampleembodiments, the one or more processors or processor-implemented modulesmay be distributed across a number of geographic locations.

Unless specifically stated otherwise, discussions herein using wordssuch as “processing,” “computing,” “calculating,” “determining,”“presenting,” “displaying,” or the like may refer to actions orprocesses of a machine (e.g., a computer) that manipulates or transformsdata represented as physical (e.g., electronic, magnetic, or optical)quantities within one or more memories (e.g., volatile memory,non-volatile memory, or a combination thereof), registers, or othermachine components that receive, store, transmit, or displayinformation.

As used herein any reference to “one embodiment” or “an embodiment”means that a particular element, feature, structure, or characteristicdescribed in connection with the embodiment is included in at least oneembodiment. The appearances of the phrase “in one embodiment” in variousplaces in the specification are not necessarily all referring to thesame embodiment.

Some embodiments may be described using the expression “coupled” and“connected” along with their derivatives. For example, some embodimentsmay be described using the term “coupled” to indicate that two or moreelements are in direct physical or electrical contact. The term“coupled,” however, may also mean that two or more elements are not indirect contact with each other, but yet still co-operate or interactwith each other. The embodiments are not limited in this context.

As used herein, the terms “comprises,” “comprising,” “includes,”“including,” “has,” “having” or any other variation thereof, areintended to cover a non-exclusive inclusion. For example, a process,method, article, or apparatus that comprises a list of elements is notnecessarily limited to only those elements but may include otherelements not expressly listed or inherent to such process, method,article, or apparatus. Further, unless expressly stated to the contrary,“or” refers to an inclusive or and not to an exclusive or. For example,a condition A or B is satisfied by any one of the following: A is true(or present) and B is false (or not present), A is false (or notpresent) and B is true (or present), and both A and B are true (orpresent).

In addition, use of the “a” or “an” are employed to describe elementsand components of the embodiments herein. This is done merely forconvenience and to give a general sense of the description. Thisdescription, and the claims that follow, should be read to include oneor at least one and the singular also includes the plural unless it isobvious that it is meant otherwise.

This detailed description is to be construed as an example only and doesnot describe every possible embodiment, as describing every possibleembodiment would be impractical, if not impossible. One could implementnumerous alternate embodiments, using either current technology ortechnology developed after the filing date of this application.

What is claimed:
 1. A computer-implemented method to analyze medicalimage data, the method comprising: obtaining, using one or moreprocessors, the medical image data comprising a plurality oftwo-dimensional (2D) medical images; performing, using the one or moreprocessors, target tissue detection and target tissue segmentation foreach 2D medical image to produce a set of segmented target tissueimages; generating, using the one or more processors, athree-dimensional (3D) model of detected and segmented target tissue;identifying, using the one or more processors, a master set of radiomicfeatures for the 3D model of detected and segmented target tissue;comparing, using the one or more processors, at least some of theradiomic features in the master set to identify redundant radiomicfeatures for the 3D model of detected and segmented target tissue;excluding, using the one or more processors, the redundant radiomicfeatures from the 3D model of detected and segmented target tissue; andextracting, using the one or more processors, a selected set of radiomicfeatures from the 3D model of detected and segmented target tissue. 2.The method of claim 1, wherein the target tissue is tumor tissue.
 3. Themethod of claim 2, wherein the tumor tissue comprises breast cancertissue, colon cancer tissue, gastric cancer tissue, endometrium cancertissue, ovarian cancer tissue, hepatobiliary tract cancer tissue,urinary tract cancer tissue, lung cancer tissue, brain cancer tissue, orskin cancer tissue.
 4. The method of claim 1, wherein the target tissueis tumor tissue and surrounding tissue.
 5. The method of claim 1,further comprising: identifying from among the selected set of radiomicfeatures, radiomic features that are significantly associated withclinical outcomes and/or genomic data.
 6. The method of claim 5, furthercomprising: identifying from among the selected set of radiomicfeatures, the radiomic features that are significantly associated withclinical outcomes and/or genomic data using a Cox proportional hazardsmodel.
 7. The method of claim 6, wherein the radiomic featuressignificantly associated with clinical outcomes and/or genomic data havea p-value <0.005 using the Cox proportional hazards model.
 8. The methodof claim 6, wherein the radiomic features are shape-based features. 9.The method of claim 6, wherein the radiomic features are intensity-basedfeatures.
 10. The method of claim 6, wherein the radiomic features aretextural features.
 11. The method of claim 6, wherein the radiomicfeatures are filter-based features.
 12. The method of claim 1, furthercomprising identify redundant radiomic features of the 3D model using apair-wise comparison.
 13. The method of claim 1, further comprising:performing the method of claim 1 (i) at a first time period before atumor therapy and again (ii) at a second time period after the tumortherapy.
 14. The method of claim 13, further comprising: determiningchanges in radiomic features from the first time period before the tumortherapy to the second time period after the tumor therapy.
 15. Themethod of claim 14, wherein the therapy is a chemotherapy.
 16. Themethod of claim 14, wherein the therapy is a radiation.
 17. The methodof claim 14, wherein the therapy is an immunotherapy.
 18. The method ofclaim 14, wherein the therapy is a poly ADP ribose polymerase (PARP)inhibitors therapy.
 19. The method of claim 14, wherein the therapy is aCAR T-cell therapy.
 20. The method of claim 14, wherein the therapy is acancer vaccine.
 21. The method of claim 14, further comprising:determining efficacy of the therapy from the changes in radiomicfeatures from the first time period before the tumor therapy to thesecond time period after the tumor therapy.
 22. The method of claim 21,further comprising: determining a next therapy in response todetermining the efficacy of the therapy.
 23. A computer-implementedmethod for generating radiomic features for use in a 3D model of atumor, the method comprising: performing, using the one or moreprocessors, using a convolution neural network, model target tissuedetection and target tissue segmentation for a plurality of 2D medicalimages to produce a set of segmented target tissue images; generating,using the one or more processors, a 3D model of detected and segmentedtarget tissue from a plurality of 2D medical images; identifying, usingthe one or more processors, using a convolution neural network, a masterset of radiomic features for the 3D model of detected and segmentedtarget tissue; identifying, using the one or more processors, using astatistical model, radiomic features that are significantly associatedwith clinical outcomes and/or genomic data; and storing or displayingthe significantly associated radiomic features in an enhanced 3D modelor in a digital report.
 24. A computing device comprising: one or moreprocessors; a user interface; and a computer-readable memory coupled tothe one or more processors, the memory storing instructions that causethe one or more processors to: perform target tissue detection andtarget tissue segmentation for each of a plurality of 2D medical imageand produce a set of segmented target tissue images; generate a 3D modelof detected and segmented target tissue; identify a master set ofradiomic features for the 3D model of detected and segmented targettissue; compare at least some of the radiomic features in the master setto identify redundant radiomic features for the 3D model of detected andsegmented target tissue; exclude the redundant radiomic features fromthe 3D model of detected and segmented target tissue; extract a selectedset of radiomic features from the 3D model of detected and segmentedtarget tissue; and store or display the extracted set of radiomicfeatures in an enhanced 3D model or in a digital report.
 25. Thecomputing device of claim 24, wherein the memory stores instructionsthat cause the one or more processors to: analyze the selected set ofradiomic features and identify from the selected set of radiomicfeatures, radiomic features that are significantly associated withclinical outcomes and/or genomic data.
 26. The computing device of claim25, wherein the memory stores instructions that cause the one or moreprocessors to: identify the significantly associated radiomic featuresusing a Cox proportional hazards model, wherein the significantlyassociated radiomic features have a p-value<0.005.
 27. The computingdevice of claim 25, wherein the memory stores instructions that causethe one or more processors to: identify the significantly associatedradiomic features correlated to tumor treatment effectiveness.
 28. Acomputer-implemented method to analyze medical image data, the methodcomprising: obtaining, using one or more processors, a first pluralityof 2D medical images captured at a first time period, performing, usingthe one or more processors, target tissue detection and target tissuesegmentation on each of the first plurality of 2D medical images toproduce a set of segmented target tissue images, and generating, usingthe one or more processors, a first 3D model of detected and segmentedtarget tissue; identifying, using the one or more processors, set ofradiomic features in the first 3D model; obtaining, using one or moreprocessors, a second plurality of 2D medical images captured at a secondtime period different from the first time period, performing, using theone or more processors, target tissue detection and target tissuesegmentation for each the second plurality of 2D medical images toproduce a set of segmented target tissue images, and generating, usingthe one or more processors, a second 3D model of detected and segmentedtarget tissue; identifying, using the one or more processors, the set ofradiomic features in the second 3D model; and comparing, using the oneor more processors, the set of radiomic features in the first 3D modeland the second 3D model and determining a change in the set of radiomicfeatures.
 29. The method of claim 28, wherein first time period isbefore a tumor therapy and the second time period is after the tumortherapy.
 30. The method of claim 29, further comprising determining atumor treatment effectiveness from the comparison of the set of radiomicfeatures in the first 3D model and the second 3D model.
 31. The methodof claim 28, wherein identifying the set of radiomic features in thefirst 3D model and/or identifying the set of radiomic features in thesecond 3D model comprises: identifying a master set of radiomicfeatures; excluding redundant radiomic features from the master set ofradiomic features; and extracting a selected set of radiomic featuresfrom the first 3D model and/or the second 3D model.
 32. The method ofclaim 28, wherein identifying the set of radiomic features in the first3D model and/or identifying the set of radiomic features in the second3D model comprises: identifying a master set of radiomic features;extracting a selected set of radiomic features from the first 3D modeland/or the second 3D model; and identifying from among the selected setof radiomic features, radiomic features that are significantlyassociated with clinical outcomes and/or genomic data.